Papers with mean average precision
TextGraphs 2019 Shared Task on Multi-Hop Inference for Explanation Regeneration (D19-53)
Copied to clipboard
| Challenge: | Detailed extended analyses of all submitted systems showed large relative improvements in accessing the most challenging multi-hop inference problems, while absolute performance remains low. |
| Approach: | The Shared Task on Multi-Hop Inference for Explanation Regeneration asks participants to regenerate detailed gold explanations for elementary science questions by selecting facts from a knowledge base of semi-structured tables. |
| Outcome: | The top-performing system achieved a mean average precision of 0.56 . the task combines facts from a knowledge base and supervised training data . |
ASU at TextGraphs 2019 Shared Task: Explanation ReGeneration using Language Models and Iterative Re-Ranking (D19-53)
Copied to clipboard
| Challenge: | Explanation Regeneration task is an intermediate step towards general multi-hop inference on large graphs. |
| Approach: | They propose a system that performs multi-hop inference and ranks a set of explanatory facts for a given elementary science question and correct answer pair. |
| Outcome: | The proposed system secured 2nd rank in the text graphs 2019 shared task with a mean average precision (MAP) of 41.3% on the test set. |
Chains-of-Reasoning at TextGraphs 2019 Shared Task: Reasoning over Chains of Facts for Explainable Multi-hop Inference (D19-53)
Copied to clipboard
| Challenge: | EMNLP 2019 shared task on 'Multi-hop Inference Explanation Regeneration' identifies chains of facts relevant to explain an answer to an elementary science examination question. |
| Approach: | They propose a system that identifies chains of facts relevant to explain an answer to an elementary science examination question. |
| Outcome: | The proposed system outperforms the second best system by 14.95 points on the mean average precision (MAP) metric. |
LEVEN: A Large-Scale Chinese Legal Event Detection Dataset (2022.findings-acl)
Copied to clipboard
Feng Yao, Chaojun Xiao, Xiaozhi Wang, Zhiyuan Liu, Lei Hou, Cunchao Tu, Juanzi Li, Yun Liu, Weixing Shen, Maosong Sun
| Challenge: | Existing legal event detection datasets only cover incomprehensive event types and have limited annotated data. |
| Approach: | They present a large-scale Chinese legal event detection dataset . they use legal events as side information to promote downstream applications . |
| Outcome: | The proposed method improves 2.2 points precision in low-resource judgment prediction and 1.5 points precision for unsupervised case retrieval. |
CLIReval: Evaluating Machine Translation as a Cross-Lingual Information Retrieval Task (2020.acl-demos)
Copied to clipboard
| Challenge: | evaluating machine translation (MT) with cross-lingual information retrieval is relatively time-consuming and subjective. |
| Approach: | They propose a toolkit that evaluates machine translation with a proxy task of cross-lingual information retrieval. |
| Outcome: | The proposed toolkit is based on the "metrics shared task" of WMT2019. |
Reasoning Over Paths via Knowledge Base Completion (D19-53)
Copied to clipboard
| Challenge: | Existing methods to predict missing links in knowledge graphs are lacking. |
| Approach: | They propose a method to automatically rank paths between a source and target entity pair using a knowledge base completion model. |
| Outcome: | The proposed method can rank and rank paths in biomedical knowledge graphs with a KBC model. |
Automated Generation of Storytelling Vocabulary from Photographs for use in AAC (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for generating context-related vocabulary from photographs are lacking for enhancing social interaction support. |
| Approach: | They propose a method for generating context-related vocabulary from photographs . their results provide insights for system optimization and usage . |
| Outcome: | The proposed method generates vocabulary with 4.6 gain in average precision . it is compared to baseline generation of English words and 6.9 for photos with correct information. |
Visual Detection with Context for Document Layout Analysis (D19-1)
Copied to clipboard
| Challenge: | a challenge in scientific literature mining is the difficulty of extracting high-quality text from formatted PDFs. |
| Approach: | They propose a method to visually segment key regions of scientific articles using object detection augmented with contextual features. |
| Outcome: | The proposed method improves the accuracy of the proposed method and the speed of the dataset. |
Did the Models Understand Documents? Benchmarking Models for Language Understanding in Document-Level Relation Extraction (2023.acl-long)
Copied to clipboard
| Challenge: | Document-level relation extraction (DocRE) models achieve consistent performance gains in DocRE, but their underlying decision rules are still understudied. |
| Approach: | They propose to use annotations to provide rationales for document-level relation extraction (DocRE) they then propose to apply a method to evaluate models' reasoning capabilities . |
| Outcome: | The proposed models exhibit different reasoning processes in contrast to humans . the proposed models render models more trustworthy and robust to be deployed in real-world scenarios. |
Distillation-Resistant Watermarking for Model Protection in NLP (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing protection methods such as watermarking only work for images but are not applicable to text. |
| Approach: | They propose a technique that injects watermarks into the victim’s prediction probability corresponding to a secret key and is able to detect such a key by probing a suspect model. |
| Outcome: | The proposed technique detects stealing suspects at 100% accuracy on four NLP tasks while the prior method fails on two. |
LGAR: Zero-Shot LLM-Guided Neural Ranking for Abstract Screening in Systematic Literature Reviews (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for abstract screening focus on binary classification settings; existing question answering (QA) based ranking approaches suffer from error propagation. |
| Approach: | They propose a systematic literature review (SLR) method that uses large language models to evaluate the SLR's inclusion and exclusion criteria. |
| Outcome: | The proposed method outperforms existing question answering (QA) based methods by 5-10 pp. in mean precision. |
Dial2vec: Self-Guided Contrastive Learning of Unsupervised Dialogue Embeddings (2022.emnlp-main)
Copied to clipboard
| Challenge: | Dialogue embeddings are a critical prerequisite for semantically understanding dialogues. |
| Approach: | They propose a self-guided contrastive learning approach called dial2vec that captures interaction patterns between interlocutors and leverages them to guide the learning of the embeddings corresponding to each interlocuter. |
| Outcome: | The proposed approach achieves 8.7, 9.0, and 13.8 points absolute improvements over the strongest baseline on the three evaluation tasks respectively. |
Contrastive Error Attribution for Finetuned Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for error tracing do not detect faithfulness errors in NLG datasets. |
| Approach: | They propose a framework to identify and remove low-quality training instances that lead to undesirable outputs. |
| Outcome: | The proposed method outperforms existing methods for detecting faithfulness errors in NLG datasets. |
OpenMSD: Towards Multilingual Scientific Documents Similarity Measurement (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for finding related papers in different languages are not effective for multilingual SDSM. |
| Approach: | They propose to use Open-access Multilingual Scientific Documents to develop multilingual SDSM models that adjust and extend state-of-the-art methods for English SDSM tasks. |
| Outcome: | The proposed model outperforms baseline methods on multilingual SDSM tasks while preserving the performance of the existing methods. |